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Abstract 
There has been an increased focus on the tenets of implementation science in special education 
research, namely on the degree to which interventions are implemented as intended (i.e., fidelity 
of implementation) and factors that influence fidelity of implementation. For complex 
interventions, such as the Self-Determined Learning Model of Instruction (SDLMD), there is an 
ongoing need to refine conceptualizations of fidelity of implementation the factors that influence 
it in inclusive educational contexts. Therefore, the purpose of this paper is two-fold: (a) to 
describe a framework for conceptualizing fidelity of implementation factors that influence 
fidelity when evaluating a complex intervention like the SDLMI and (b) to present initial content 
validity and internal consistency of a new measure designed to assess specific aspects of fidelity 
of implementation of the SDLMI in inclusive, general education classrooms. Implications and 
future directions for research in implementation science, special education, and self- 
determination interventions are described. 
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The Self-Determined Learning Model of Instruction: Promoting Implementation Fidelity 

The Self-Determined Learning Model of Instruction (SDLMI; Shogren et al., 2018d; 
Wehmeyer et al., 2000) is an evidence-based intervention to promote self-determination and 
post-school outcomes (National Technical Assistance Center on Transition, 2016). The SDLMI 
is unique as it was not designed to be a curriculum, but instead a model of instruction (Joyce & 
Weil, 1980; Shogren et al., 2018d). Models of instruction provide a framework to organize 
curriculum and instruction. The SDLMI provides trained facilitators (e.g., general and special 
educators) with a structured process to engage students in learning to self-direct the goal setting 
and attainment process across curricular areas. As such, the SDLMI was designed to be overlaid 
on any curricular content (e.g., mathematics, science, art, transition planning) to support students 
to self-direct the process of setting and attaining goals, building key 21st century and career and 
college readiness skills (Morningstar et al., 2017; National Research Council, 2012) while also 
enhancing engagement in content-based learning (Shogren et al., 2015c). 

Implementation of a model of instruction like the SDLMI introduces complexity as it 
necessitates problem solving and flexibility on the part of the facilitator (e.g., general or special 
educator) to customize to their particular curricular area and classroom context as well as the 
delivery of specific supports for students to generalize learning outside of targeted instruction. 
In the case of the SDLMI, educators must devote instructional time to teaching the goal setting 
and attainment process defined by SDLMI implementation protocols, aligning this instruction 
with the focus of their curriculum and supporting students to apply taught goal setting and 
attainment skills as they are engaged in the curriculum (Burke et al., 2019; Raley et al., 2018). 

The SDIMI is divided into three distinct phases - Phase 1: Set a Goal, Phase 2: Take 


Action, and Phase 3: Adjust Goal or Plan — of implementation that are repeated over time. 
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Facilitators are trained to use the SDLMI to support students in answering a set of 12 Student 
Questions (four per phase) to guide themselves through a process of setting individualized goals 
for learning and then implementing self-identified steps to achieve those goals. The targeted 
outcome of SDLMI instruction is enhanced student self-direction and self-regulation in the goal 
setting and attainment process, and the teacher facilitator shapes learning experiences to lead to 
desired outcomes. In inclusive, general education SDLMI implementation, teacher facilitators 
are trained to embed brief and explicit instruction (i.e., 15-minute “mini-lessons”) on the SDLMI 
Student Questions, targeting a specific class-related goal (e.g., academic skills, self-management 
skills, learning preparatory skills) at least twice a week. Teacher facilitators then prompt and 
support students to implement strategies they have learned during SDLMI “mini-lesson” 
instruction throughout the rest of learning activities in a class period to promote generalization 
and use of the self-regulated goal-setting and problem-solving skills. Therefore, teaching the 12 
Student Questions, which is guided by standardized Teacher Objectives and Educational 
Supports, drive the instruction delivered (see Raley et al., 2018 for further information). 

This delicate balance between structured yet flexible implementation introduces unique 
considerations for assessing fidelity as well as documenting factors influencing fidelity to 
SDLMI instruction, consistent with any complex intervention (Graham et al., 2016; Harn et al., 
2013). The field of implementation science has provided guidance for utilizing evidence-based 
interventions in natural contexts (e.g., school systems) to illustrate how interventions or 
programs are put into practice to produce valued outcomes (Fixsen et al., 2005). However, in 
order to fully and effectively use interventions in natural contexts, leaders in implementation 
science stress a need for increased focus on (a) documenting if interventions are implemented as 


intended (i.e., implementation fidelity) and (b) operationalizing factors that influence 
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implementation and accounting for these factors both in planning for and analyzing 
implementation and its impact on outcomes (Cook & Odom, 2013; Fixsen et al., 2010). 

The critical role of documenting, planning, and accounting for fidelity of implementation 
and the factors that influence it have been noted in several efficacy trials of the SDLMI (Shogren 
et al., 2018a; Shogren et al., 2012; Wehmeyer et al., 2012). For example, differential outcomes 
have been noted across disability groups and across different content-area implementations (i.e., 
academic versus transition planning goals for students with intellectual disability versus students 
with learning disabilities; Shogren et al., 2012). The noted differences in outcomes suggest 
student characteristics such as (e.g., disability label, gender, age) and implementation factors 
(e.g., teacher training, classroom characteristics) likely interact to influence outcomes. Other 
research has documented that teacher self-reported fidelity of the SDLMI influences not only 
student self-determination outcomes, but is also influenced by students’ baseline levels of self- 
determination (Shogren et al., 2019a), suggesting an interaction of student, teacher, and 
implementation factors. Given the consistent finding that there is variability in outcomes of 
SDLMI intervention that is unexplained, there is a need for ongoing attention to analyzing 
fidelity of implementation and factors that influence it. 

Given these areas of need, the purpose of this paper is two-fold. Building on a review of 
existing fidelity frameworks that describe implementation considerations related to fidelity and 
the factors that influence it, we first describe a broad framework for conceptualizing fidelity of 
implementation and influencing factors for the SDLMI. Next, we build on one aspect of the 
framework — directly assessing the fidelity of implementation of the SDLMI by teachers in 
inclusive, general education classrooms — and finally describe the development, initial content 


validity and internal consistency of the SDLMI Fidelity Measure: Inclusive, General Education 
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Version. We conclude with future directions for conceptualizing fidelity of implementation 
broadly, and highlight implications for designing, implementing, and evaluating the impact of 
complex interventions like the SDLMI on student outcomes. 
Understanding Fidelity: Existing Frameworks 

The question of whether or not an intervention was implemented as intended can initially 
seem to be a simple concept to define, measure, and integrate into analyses of the impacts of 
interventions on outcomes. However, there is no consensus definition of fidelity, elements of 
fidelity to be measured in intervention research, or the most appropriate way to integrate 
information on fidelity and factors that influence it into (a) planning for implementation and (b) 
analyses of intervention efficacy or effectiveness. Multiple frameworks have been introduced in 
disability, education, and related fields (Harn et al., 2013). There is growing consensus that 
multidimensional approaches are necessary, particularly for complex interventions (Graham et 
al., 2016; Harn et al., 2013), but there remains substantial lack of agreement on how to define the 
fidelity construct and how to differentiate factors influencing fidelity from observable, 
measurable aspects of implementation in a classroom. 

Dane and Schneider (1998) identified five dimensions of fidelity that have been widely 
used to guide the development of fidelity of implementation measures and indices: (1) 
adherence, (2) dose, (3) quality, (4) participant responsiveness, and (5) program differentiation. 
These five elements, perhaps with the exception of program differentiation, can be aligned with 
the other frameworks that adopt a structural (adherence and dose) and process (quality and 
responsiveness) dimensions to conceptualize fidelity (O’Donnell, 2008). Other frameworks, 
however, are broader and focus not only on observing fidelity of implementation — although, this 


remains one critical element — but also conceptualizing factors that influence implementation 
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outside of the actual instructional process. Arguments for broader frameworks center around the 
need for more robust understandings of implementation to enhance the internal and external 
validity of the conclusions drawn about the relationship between the intervention as implemented 
and outcomes. For example, the National Institutes of Health (NIH) Behavior Change 
Consortium (Bellg et al., 2004) suggested fidelity should be conceptualized as an essential part 
of study design and planning and should be monitored throughout the course of an intervention. 
Bellg et al. (2004) described five areas of fidelity and influencing factors to be measured: (a) 
study design (e.g., the design enables testing of the hypotheses, which is aligned with theoretical 
foundations; the intervention can be delivered at the same dose to each participant), (b) training 
of implementers (e.g., training implementers and measuring skill acquisition to ensure that skills 
do not drift over time across implementers), (c) delivering treatment (e.g., monitoring 
implementation and adherence to the implementation protocol, controlling for non-treatment 
effects, minimizing contamination), (d) quantifying receipt of treatment (e.g., monitoring how 
able participants are to perform intervention-related skills), and (e) enactment of treatment skills 
(e.g., how well do skills generalize to non-treatment settings). 

Carroll et al. (2007) attempted to further differentiate fidelity and influencing factors, 
defining adherence as the key aspect of the measurement of fidelity stating, “if an implemented 
intervention adheres completely to the content, frequency, duration, and coverage prescribed by 
its designers then fidelity can be said to be high” (p. 4). Carroll et al. (2007) asserted other 
factors often included in fidelity assessment are instead moderators of the relationship between 
the intervention and adherence, including intervention complexity, facilitation strategies (i.e., 
supports used to standardize implementation such as manuals or trainings), quality of delivery 


(i.e., extent to which the intervention is delivered in a way to achieve the intended results), and 
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participation responsiveness (i.e., participants’ views on relevance of the intervention to them). 
These factors could be referred to as process elements or more broadly related to study design 
and implementation as defined by the NIH Behavior Change Consortium. Carroll et al. (2007) 
asserted these factors should be utilized to explore the reasons why an intervention was not 
implemented as intended (e.g., adherence was low) to inform future implementation and support 
strategies, thereby necessitating their measurement. 

Highlighting that there are unique issues to consider in defining fidelity and factors that 
influence it for complex interventions, Graham et al. (2016) defined three elements to guide a 
review of the literature to inform their development of fidelity measures for a self-management 
intervention for adults with disabilities. They focused on content of the intervention (e.g., topics 
or components covered), delivery details (e.g., formats used, contact with interventionists), and 
intervention quality/competence of delivery (e.g., training, supervision). They also highlighted 
the role of individualization to the needs of participants. Graham et al. (2016), unlike Carroll et 
al. (2007), specifically included issues related to quality/competence of delivery to their 
definition of fidelity as assessed, not simply as factors that influenced fidelity, given the 
criticality of this dimension in complex interventions necessitating changes both in facilitators 
and participants. In reviewing the research, they found that studies tended to describe content 
relatively well, but there was limited measurement of delivery beyond basic adherence (e.g., 
number of intervention sessions delivered). Quality/competence were also frequently described 
in research reviewed by Graham et al. (2016) by detailing training provided for implementers, 
but few formal measures of quality of delivery or alignment with protocols were included. 
Findings from Graham et al. (2016) also suggest fidelity measures rarely address issues related to 


complex interventions that are influenced not only by intervention delivery by facilitators (e.g., 


SDLMI FIDELITY OF IMPLEMENTATION 8 


teachers) but also by participant adherence, responsiveness, and use of taught strategies. 
Fidelity and the Self-Determined Learning Model of Instruction 

The conceptualizations of fidelity described in the previous section highlight variability 
in how fidelity is conceptualized and the need for a clear framework of how to assess fidelity in 
the classroom and document/plan for the factors that influence it. We posit that a framework for 
a complex intervention like the SDLMI should be multidimensional and address factors that 
influence fidelity as well as the direct assessment of fidelity of implementation. We aligned key 
dimensions from multiple frameworks to develop a fidelity framework for the SDLMI when it is 
implemented in inclusive, general education classrooms. We identified six key dimensions; see 
the left column of Table 1. The framework encompasses (a) factors that influence fidelity (e.g., 
study design, training providers), (b) elements that could be specifically measured through 
fidelity observations (e.g., adherence, quality of delivery, participant responsiveness), and (c) 
factors related to the enactment of treatment skills and program differentiation. We considered 
how each dimension in the framework specifically applied to SDLMI implementation, drawing 
on information from SDLMI implementation protocols developed to enable implementation in 
inclusive, general education classes (Shogren et al., 2019c). 
Program Differentiation 

The first dimension relates to how the intervention is differentiated from other 
interventions and practices adopted in the context, which are similarly referred to as study design 
and intervention complexity across other fidelity frameworks. With regards to study or 
intervention complexity, when SDLMI fidelity is assessed as part of an efficacy or effectiveness 
trial or in the context of implementation evaluation in inclusive, general education settings, there 


are several critical factors to consider. One is the degree to which there is a clearly aligned 
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theoretical framework that informs a theory of change that defines process and outcome 
measures. The SDLMI was designed to align with Causal Agency Theory (Shogren et al., 
2015c), which provides a theoretical framework for the development of self-determination in 
supportive contexts. As such, the SDLMI has a theoretical foundation that leads to a clear theory 
of change, namely that when skills associated with self-determination (i.e., executive processes 
including goal setting, self-regulation, inhibitory control, and behavioral flexibility) are taught in 
contexts (e.g., inclusive, general education classrooms) that are supportive of the development of 
student autonomy, growth in self-determination and goal attainment will occur in the short term, 
with longer-term benefits across diverse life domains (e.g., academic learning, postschool 
employment, community participation). Given that these areas can be assessed using reliable 
measures (see subsequent receipt of treatment/enactment of treatment skills section) in the short- 
term during implementation, hypotheses about the development of student self-determination can 
be meaningfully assessed. 

The SDLMI has clear distinctions from other interventions to enhance academic progress 
(e.g., class benchmarks, unit accrual) and postschool transition-related outcomes (e.g., 
employment skill development and experiences) and as a complex intervention, it requires 
teacher facilitators to problem solve throughout implementation, aligning with the curriculum 
they are teaching. To ensure the core components of the model (i.e., Student Questions, Teacher 
Objectives, Educational Supports) are taught; teacher facilitators build Implementation 
Schedules for SDLMI content, planning for how to deliver targeted lessons on skills associated 
with self-determination as well as how to infuse learning throughout other instruction. A 
standardized Implementation Schedule format is utilized to document adherence and ensure that 


the core components of the SDLMI are delivered throughout content instruction in a systematic 
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way, again differentiating this from other curriculum-driven models. 

We have also developed and are utilizing in large-scale randomized controlled trials 
(RCTs) of the SDLMI, a School Profile Form that allows us to capture other interventions that 
are implemented in schools. The research team collaborates with participating school members 
(e.g., administrators, teachers) to complete the School Profile Form during a structured 
interview. Items on the School Profile Form ask school members about other school initiatives 
that might have similar activities to those associated with the SDLMI. Thus, the Implementation 
Schedule, which is continually updated as teacher schedules change, and the annual School 
Profile Form can be used in studies to ensure adherence and program differentiation. 

Training Providers 

Similarly, systematically training SDLMI facilitators is also critical to ensure that SDLMI 
implementation occurs as intended. As shown in Table 1, several steps have been taken to create 
standardized training materials to promote adherence to the SDLMI. First, standardized three- 
day training materials have been developed to prepare general and special education teachers for 
SDLMI implementation in inclusive, general education classrooms, informed by training and 
implementation supports for other complex interventions (Cheney et al., 2010; Lane et al., 2018; 
Shogren et al., 2015a). The SDLMI Teacher’s Guide (Shogren et al., 2018d) and associated 
mini-lessons (i.e., one semester’s worth of bi-weekly 15-minute lessons targeting specific 
Student Questions) have been developed and a set of online modules that can be shared twice a 
month with facilitators to guide implementation and troubleshoot problems are available for 
implementing schools. Each tool defines the parameters of SDLMI implementation and 
standardizes both the training and ongoing supports and resources to promote fidelity. 


Additionally, the Teacher Self-Determination Knowledge, Skills, and Use Survey (SD- 
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KSU) (Shogren et al., 2018b) was adapted from the Knowledge, Confidence, and Use Surveys 
used in other professional learning series (Lane et al., 2015; Oakes et al., 2018), and the SDLMI 
Professional Development Survey (Shogren et al., 2018c) was created to evaluate teacher 
learning during standardized SDLMI professional development training and over the course of 
implementation. For example, in a current RCT, teacher facilitators complete the SD-KSU and 
the SDLMI Professional Development Survey before annual trainings to provide a baseline of 
teacher knowledge, skills, and use of the SDLMI. Teachers then complete the SD-KSU at an 
additional timepoint at the end of the implementation year to explore changes that occur as a 
result of the training and experience implementing the SDLMI over an academic year. 
Preliminary analyses have suggested teachers improve in their knowledge of self-determination 
and associated after the professional development training (Bojanek et al., 2020). 
Adherence 

Beyond the Implementation Schedules already described and the SD-KSU which is 
utilized to document changes in teacher use of the SDLMI after training and following 
implementation, it is necessary to have a standardized tool to support (a) teachers in self- 
monitoring their implementation, (b) SDLMI coaches in reporting observations and providing 
feedback to teachers, and (c) external observers in documenting fidelity of the implementation of 
the SDLMI. A standardized fidelity tool that can be used to collect, report, and analyze the 
degree to which the intervention is implemented as intended is one of the most widely 
recognized and evaluated indices of fidelity. Reporting on fidelity of implementation is a best 
practice in intervention research (Gersten et al., 2005). For this reason, the SDLMI Fidelity 
Measure: Inclusive, General Education Version (Shogren et al., 2018f) was developed to be used 


by SDLMI coaches conducting classroom observations of implementation and/or by external, 
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blinded observers to document fidelity. This tool, its development, and preliminary 
psychometrics will be described in later in this manuscript; however, it is important to note that 
the tool provides data on three of the dimensions of fidelity described in Table 1: adherence, 
quality of program delivery, and participant responsiveness. 

We developed a parallel version, the SDLMI Teacher Fidelity Measure: Inclusive, 
General Education Version, that provides teachers with an opportunity to self-monitor and report 
their perceptions of their fidelity with the research and SDLMI coaching team. Utilizing the 
SDLMI Fidelity Measure: Inclusive, General Education Version in conjunction with the SDLMI 
Teacher Fidelity Measure can, for example, allow for examination of the congruence of 
perceptions of implementation by the coach/external observer and the implementing teacher, 
particularly when the observation is part of the SDLMI Coaching Model. 

Quality of Program Delivery 

As mentioned, quality of program delivery as well as degree of individualization are both 
central to SDLMI implementation, particularly as the intervention should be individualized to 
student and class learning needs. In addition to data collected from the teacher, coach, and/or 
observer using the SDLMI Fidelity Measure and SDLMI Teacher Fidelity Measure, quality of 
program delivery data can also be collected from the SDLMI Coaching Conversation Notes, 
which is part of the standardized SDLMI Coaching Model (Hagiwara et al., 2020). The SDLMI 
Coaching Model includes standardized processes for classroom observations and coaching 
conversations. The SDLMI Coaching Conversation Notes outlines the topics to review during 
the coaching session and includes a place for coaches to record important information for future 
reference, such as the goals and action steps decided upon to ensure a productive and meaningful 


conversations with teachers during subsequent coaching sessions. Further, discrepancies that 
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emerge between SDLMI fidelity observers and teacher self-report of fidelity can be discussed. 
Additionally, the Use ratings from the Teacher SD-KSU survey can be used to indicate enhanced 
quality in program delivery as changes in teacher use of components of self-determination are 
observed over time. Questions about growth in teacher skills over time and its association with 
additional implementation experiences can be examined, as well as the relationship between 
quality of program delivery/degree of individualization and outcomes. 
Participant Responsiveness 

The responsiveness of participants is also critical to assess as part of the implementation 
of the SDLMI, as students must engage in the self-regulated, problem-solving and goal-setting 
process that is taught by teachers using the SDLMI. If participants are not engaged in the 
process, they cannot move through the process and utilize the skills taught to engage in goal- 
directed action and ultimately enhance self-determination and postschool outcomes. Therefore, 
this dimension is directly rated on the teacher, coach, and observer versions of the SDLMI 
Fidelity Measure: Inclusive, General Education Version and SDLMI Teacher Fidelity Measure 
with items that directly assess student responsiveness during SDLMI lessons. For example, 
using a five-point rating scale, observers and coaches rate the degree to which students respond 
to the teacher’s questions or prompts during the SDLMI lessons and the content instruction. 
These items, in addition to the teacher-reported information from the SDLMI Teacher Fidelity 
Measure, provide information on fidelity as it relates to participant responsiveness. Gathering 
information directly from students using the Self-Determination Inventory: Student Report 
(SDI:SR) (Shogren & Wehmeyer, 2017) documents the changes resulting from the intervention. 
Receipt of Treatment 


Beyond looking at participant responsiveness to the intervention, it is also important to 
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monitor the degree to which students are using the skills taught throughout the intervention, 
thereby creating direct linkages between fidelity and outcomes. This both relates to the ability of 
students to immediately use the skills targeted for change in the short-term, as well as 
generalization across contexts. For example, the Access Survey (Shogren et al., 2018e) can be 
used to measure changes in access to the general education curriculum of students with 
disabilities, which is a hypothesized outcome of SDLMI instruction that has been demonstrated 
in previous trials (Shogren et al., 2012). To measure distal student outcomes, data on postschool 
outcomes is utilized, as research has shown that enhanced self-determination for students with 
disabilities leads to enhanced postschool outcomes (Shogren et al., 2015d). Additionally, the 
SDLMI Fidelity Measure: Inclusive, General Education Version, SDLMI Teacher Fidelity 
Measure, and Teacher SD-KSU provide information on changes in teachers’ implementation as 
they develop fluency with the SDLMI. Given that the SDLMI is a model of instruction that is 
intended to change the way that teacher facilitators deliver instruction to their students, 
measuring changes in teacher knowledge, skills, use, and fidelity of implementation also serve as 
indicators of receipt of treatment from facilitators. 
SDLMI Fidelity Measure: Inclusive, General Education Version 

As noted, direct observations of implementation of a complex intervention like the 
SDLMI are central to any fidelity framework and reporting data on fidelity is necessary to 
contextualize the relationship between an intervention and outcomes. Based on the literature, we 
identified dimensions that should be assessed through objective, external observations of 
implementation of the SDLMI in inclusive, general education contexts, namely: adherence, 
quality of program delivery, and participant responsiveness. To develop the SDLMI Fidelity 


Measure: Inclusive, General Education Version, we began with previous fidelity measures, 
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including a measure developed with the National Research Implementation Network to assess 
SDLMI implementation in transition planning with students with disabilities, the SDLMI Fidelity 
Measure: Transition Planning (Shogren, Wehmeyer, Antosh, et al., 2015b) as well as emerging 
guidelines for developing measures for examining implementation of complex interventions 
(Martinez et al., 2014). The SDLMI Fidelity Measure: Transition Planning focused on two 
dimensions of fidelity (adherence and quality of program delivery) and yielded an overall fidelity 
score that has been reported in previous research (Shogren et al., 2019b). The measure was also 
slightly adapted to enable teachers to self-monitor their fidelity of implementation (Shogren et 
al., 2019a). Ratings on the measure suggested similarities in teacher self-report and external 
observers’ ratings (which ranged from 75% to 82% fidelity across the three phases of the 
SDLMI) (Shogren et al., 2019b). 

To extend the use of the SDLMI to inclusive, general education classrooms with students 
with and without disabilities and learn from the application of the measure in transition planning, 
several issues were identified. First, it became apparent that it may be useful to take a 
dimensional approach to fidelity by specifically exploring adherence, quality of program 
delivery, and participant responsiveness. This dimensional approach allows for more nuanced 
examination of the relationship between dimensions and student outcomes as well as for 
enhanced precision in providing coaching supports. Additionally, implementation in inclusive, 
general education contexts, which necessitates targeted SDLMI instruction as well as supports 
for the generalized use of SDLMI skills during core content instruction (Raley et al., 2018), 
required consideration of ways to examine fidelity during SDLMI instruction as well as during 
other instructional times when SDLMI skills were to be applied (which had not been considered 


in previous measures). To modify the measure, we consulted with an expert in implementation 
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evaluation, who helped us refine our overall fidelity framework and guided modifications. The 
refinement process began with considering when fidelity observations would occur in inclusive, 
general education settings per SDLMI implementation protocols, which resulted in two main 
sections of the SDLMI Fidelity Measure: Inclusive, General Education Version: Part B - SDLMI 
Lesson Observation (when targeted SDLMI mini-lessons are being delivered) and Part C - 
Content Instruction Observation (when SDLMI content is infused into content instruction). Part 
A simply collects background information from the teacher before the SDLMI lesson to clarify 
what will be observed, including targeted Student Question(s) and Teacher Objective(s). 

Table 2 provides excerpts from each section. The SDLMI mini-lesson observation (Part 
B) includes 12 items and the content instruction observation (Part C) includes 7 items in which 
the observer assesses the degree to which the teacher embeds opportunities for students to work 
toward their goals during content instruction without direct SDLMI instruction. Parts B and C 
each initially included questions targeting the three dimensions of fidelity: adherence, quality of 
program delivery, and participant responsiveness; however, analyses (described subsequently) 
ultimately led to adherence items being dropped from Part C: Content Instruction Observation as 
teacher facilitators did not necessarily use specific SDLMI core components (e.g., Student 
Questions, Teacher Objectives) during content instruction as the focus was on infusing 
opportunities for student use of these skills during this time. This distinction highlights the 
factors related to complex interventions that must be considered in assessing fidelity, as creating 
opportunities to enact skills may occur outside of explicit instruction. 

In Part B, adherence is measured with four items that assess the degree to which the 
teacher facilitator demonstrates core components of the SDLMI during the 15-minute lesson, 


including posing targeted Student Question(s), meeting Teacher Objective(s), using Educational 
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Support(s). Quality of program delivery includes five items that provide information on the 
teacher’s facilitation of student-directed instruction during the lesson. One of the items in the 
quality of program delivery dimension of Part B represents an overall assessment of the degree 
to which the SDLMI lesson instruction was student-directed as opposed to teacher-directed. The 
participant responsiveness dimension in Part B is comprised of three items that provide data on 
the extent to which students are engaged by and involved in the SDLMI during the lesson. 
Quality of program delivery is measured in Part C based on four items that assess the 
quality of the integration of the SDLMI into the content instruction, including the degree to 
which the teacher provides Educational Support(s) to enable students to work on goals, action 
plans, and/or evaluation strategies related to core content. Participant responsiveness or use of 
SDLMI strategies during the content instruction observation is assessed based on three items that 
focus on students’ generalization and use of strategies learned during the SDLMI lesson into the 
content instruction. After careful consideration regarding the rating scale for items, we opted to 
utilize different scales across items to accurately reflect how they could be assessed by an 
observer or coach. For example, the items related to adherence in Part B (four items) are rated 
on a three-point Likert-type scale (No, Partially, Yes) and items in the quality of program 
delivery and participant responsiveness dimensions across both sections are rated on a five-point 
Likert-type scale (in which 1 = Not at all, 2 = A little, 3 = Somewhat, 4 = Mostly, and 5 = 
Definitely). Utilizing different rating scales allowed for observers and coaches to rate items ina 
way that aligned with the associated dimension and the context in which they are measured. 
After developing the framework for the measure and potential items, several rounds of 
iterative feedback from experts in the use of the SDLMI and implementation evaluation were 


collected and used to refine the measure. One major suggestion was to incorporate open-ended, 
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written responses per item so observers can provide insight into the rating to allow us to explore 
factors that might impact reliability of ratings, as well as to inform coaching or feedback 
provided to teacher facilitators. We included question prompts in each open-ended section to 
provide observers and coaches with an idea of what they might write in the section; however, 
observers are not required to answer the specific question prompts provided for each item. 
Preliminary Data on the SDLMI Fidelity Measure: Inclusive, General Education Version 
Data were collected using the SDLMI Fidelity Measure: Inclusive, General Education 
Version by trained, blinded external observers during the first semester of implementation of the 
SDLMT in inclusive, general education classrooms as part of a large, three-year RCT currently 
being conducted in six high schools across in the Mid-Atlantic. During the semester when data 
was collected, the SDLMI was implemented by 12 general and special education teacher 
facilitators who taught English Language Arts (ELA; n = 20) or Science (n = 16) classes with 
899 students (class size ranged from 13 to 29 students, and approximately 16% [n = 144] of 
students had Individualized Education Programs). Three observations using a paper-and-pencil 
version of the tool were collected per teacher for 36 total observations. Fidelity data were 
collected by three trained observers who were retired school administrators with numerous years 
of experience conducting classroom observations in inclusive, general education classrooms. All 
fidelity observers attended a two-day training prior to the start of the school year that provided a 
comprehensive overview of self-determination and the SDLMI as well as an in-depth training on 
applying the SDLMI Fidelity Measure: Inclusive, General Education Version to example 
classroom videos of SDLMI instruction. All primary and secondary observers were required to 
reach 90% reliability across rating scale responses in order to successfully complete SDLMI 


fidelity observer training. We examined interrater reliability of 30% of observations by having a 
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secondary observer also complete the SDLMI Fidelity Measure: Inclusive, General Education 
Version simultaneously in the same classroom. Kappa ranged from 0.62 to 0.95, and using the 
criteria put forward by What Works Clearinghouse where 0.50 or greater is deemed acceptable 
(What Works Clearinghouse; WWC, 2017), we focused on the primary observers responses on 
the SDLMI Fidelity Measure: Inclusive, General Education Version in isolation in data analyses. 
Data Analysis 

Using the items from the three dimensions of fidelity described previously (i.e., 
adherence, quality of delivery, participant responsiveness), we estimated Pearson correlations to 
examine the relation among items across the sections of the SDLMI Fidelity Measure: Inclusive, 
General Education Version (i.e., Part B: SDLMI Lesson Observation, Part C: Content 
Instruction Observation). We collapsed the data across SDLMI Phase 1, 2, and 3 observations. 
This allowed us to determine if the functioning of the items depended on whether the observation 
was made from a dedicated SDLMI lesson or during the content instruction observation, 
exploring if our division of items into Parts B and C was justified. Prior to estimating the item 
level correlations, one item in the adherence dimension of Part B (“Did the teacher provide 
students with Educational Support(s)? If so, which of the following Educational Supports did the 
teacher provide?”) was dichotomously coded to indicate whether the teacher provided any 
Educational Supports (0 = No, 1 = Yes). In the event item functioning depended on section of 
the measure (i.e., Part B or C), the plan was to create dimension scores (i.e., adherence, quality 
of program delivery, participant responsiveness) within each Part by taking the sum across the 
relevant items and report them as percent of maximum (POM) scores. We then examined scale 
scores across dimensions within Parts B and C as well as broken down by the three phases of the 


SDLMI to look for differences across Parts or Phases. 
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Results 

When examining item-level correlations for the SDLMI Fidelity Measure: Inclusive, 
General Education Version, a clear pattern emerged. Across fidelity dimensions, items had 
stronger relations within sections compared with between sections. In response to this finding, 
we decided to retain the organization of the scale into Parts B and C, and moved forward with 
examining scale scores by Part, fidelity dimension, and phase of SDLMI instruction. As shown 
in Table 3, with respect to the adherence dimension in Part B, mean fidelity was 77.14 percent of 
the maximum score (SD = 22.44). With respect to the quality of program delivery dimension 
associated with Part B, mean fidelity was 65.22 with a standard deviation of 18.77. In terms of 
participant responsiveness during the SDLMI lesson (i.e., Part B), mean fidelity was 61.52 with a 
standard deviation of 19.94. Due to content instruction (Part C) being independent of explicit 
instruction on the SDLMI (Part B), no Part C items were hypothesized to be part of the 
adherence dimension, as previously mentioned. In terms of Part C quality of delivery, mean 
fidelity was 40.29 with a standard deviation of 19.63. In terms of Part C participant 
responsiveness, mean fidelity was 37.14 with a standard deviation 21.16. We then examined 
correlations among the scale scores, finding significant positive associations with the exception 
of the association between the adherence dimension of Part B and the participant responsiveness 
dimension of Part C (r = 0.31, p= 0.07). The largest associations were observed between quality 
of program delivery and participant responsiveness: 0.87 in Part B and 0.84 in Part C. Table 4 
contains a correlation matrix for all scale scores across all phases of the SDLMI. 

As shown in Table 5, when further breaking down fidelity scores across SDLMI Phases 
1, 2, and 3 for Part B, the percent of maximum points available was highest for adherence and 


ranged from 74.29 to 84.17, followed by quality of program delivery which ranged from 60.87 to 
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70.29; and finally, participant responsiveness which ranged from 58.52 to 63.89 across SDLMI 
phases. When examining the fidelity dimensions of Part C: Content Instruction Observation, the 
percent of maximum was highest for quality of program delivery which ranged from 37.50 to 
45.00; whereas, participant responsiveness ranged from 32.59 to 44.44. Regardless of the 
dimension or Part B or C, scores were highest during Phase 2. 
Implications 

Overall, preliminary analyses of the data collected over one semester of implementation 
using the SDLMI Fidelity Measure: Inclusive, General Education Version suggest that, with 
training, blinded classroom observers can use the tool reliably to observe adherence, quality of 
program delivery, and participant responsiveness during targeted, 15-minute SDLMI lessons 
(Part B) as well as observe the quality of program delivery and participant responsiveness when 
the SDLMI content is embedded throughout other non-SDLMI instruction (Part C). The results 
suggest items used to assess fidelity are best grouped separately in Parts B and C, suggesting 
differences in what is being observed across SDLMI and non-SDLMI instruction, as might be 
expected but is useful to confirm for a complex intervention like the SDLMI. As also 
anticipated, the data in Table 3 suggest that quality and responsiveness are rated as higher during 
Part B than Part C instruction. Given that other instruction is occurring during Part C, the degree 
to which SDLMI concepts and learning are emphasized during a Part C observation is likely to 
vary as well as likely be lower than during targeted instruction. Additionally, it should be noted 
teacher facilitators in the referenced RCT were in their first semester of implementing the 
SDLMI and it is possible their scores on Part C will increase over time as they gain fluency with 
the model and integration in content instruction and associated activities. Further research is 


needed to more fully examine these relationships, as well as the relationship between dimensions 
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of fidelity over time, as there was a stronger correlation between quality of program delivery and 
participant responsiveness in Part B and Part C, but a slightly lower correlation in Part B of 
quality of program delivery and participant responsiveness with adherence. This suggests there 
may be more overlap in quality of program delivery and participant responsiveness than with 
either of these dimensions and adherence, but the degree to which these patterns hold over time 
and with larger samples has yet to be examined. 

There is also a need for ongoing research to establish the necessary and expected levels 
of fidelity for implementing complex interventions, as well as the level of fidelity across 
dimensions that will impact student outcomes. For example, Borrelli et al. (2005) defined 
studies with “high treatment fidelity” in the field of health behavior research as those studies that 
had 80% or greater adherence. Therefore, the level of adherence found during Part B instruction 
with the SDLMI approached this standard (77%); however, levels of quality of program delivery 
and participant responsiveness were lower in Part B observations (65 and 62%, respectively) and 
much lower in Part C (40 and 37%, respectively). Limited work, however, has focused on 
expected levels of quality of program delivery and participant responsiveness, compared to 
adherence. The values in this study, particularly for adherence and quality of program delivery 
are similar to our previous work assessing the fidelity of the SDLMI in the context of transition 
planning, where an overall fidelity score comprised of questions of adherence and quality ranged 
from 75% to 82% across the three phases of the SDLMI (Shogren et al., 2019b). Further, in 
previous work, we established that higher teacher self-monitoring of fidelity scores were 
associated with higher self-determination outcomes, although initial student self-determination 
also influenced fidelity (Shogren et al., 2019a). Future work is needed that examines the impact 


of level of fidelity across dimensions on student outcomes, particularly examining differences 
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that might emerge over time. Additionally, other factors that might influence fidelity, including 
the other areas of our framework shown in Table 1 that are not included in the fidelity measures 
should be further examined and added to ongoing analyses. 

Also of interest was the finding depicted in Table 5. Specifically, across the three 
SDLMI Phases, which occur in temporal order over the course of a semester, there were 
differences in the ratings of adherence, quality of program delivery, and participant 
responsiveness. This is consistent with previous work that has suggested variability across 
SDLMI phases, particularly early in teacher implementation (Shogren et al., 2019b); however, 
the pattern in this study (highest in Phase 2) will require further research. For example, if there 
was a Steady increase across the three Phases (i.e., over time and implementation) this may 
suggest increased teacher fluency over time. Or, if there was initially high fidelity, followed by 
lower fidelity, there may be challenges with sustaining the intervention. However, these data 
suggest there may be factors associated with the SDLMI phases or timing. For example, is Phase 
2 is more complicated to implement than Phase 3? Or is the timing of Phase 3 at the end of the 
semester is more challenging than the timing of Phase 1 at the start of the semester? More 
research is needed, particularly longitudinal analysis that will be possible as the RCT within 
which this preliminary data was collected, that directly examines factors that influence fidelity as 
well as outcomes. 

Finally, a limitation of these preliminary fidelity findings is the absence of outcome data 
to (a) inform the relationship between fidelity and student and teacher outcome data and (b) 
explore the degree to which the fidelity dimensions targeted (adherence, quality of program 
delivery, and participant responsiveness) are related to student and teacher outcomes. Although 


data on student and teacher outcomes are collected within the larger research project, integrating 
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the current fidelity findings with outcome data was beyond the scope of the present analysis. 
Further research is needed that explores the relationship between implementation fidelity and 
outcome data to guide the development of SDLMI implementation supports and identify key 
areas to emphasize in teacher training and coaching. 
Overall Conclusions and Future Directions in Fidelity of Implementation 

As noted, while measuring whether an intervention was implemented as intended seems 
like a simple problem, it is much more complex than often acknowledged, particularly for 
interventions like the SDLMI that require (a) changes in teacher and student behaviors and (b) 
targeted instruction as well as supports for generalization. As such, rooted in an extensive 
review of the literature, we identified a framework for conceptualizing fidelity that focused on 
not only the direct assessment of implementation but also the analysis of factors that influence 
fidelity. Building on this framework, we developed a measurement tool to be used by trained 
observers as well as by trained coaches called the SDLMI Fidelity Measure: Inclusive, General 
Education Version. This measure focused on three dimensions of fidelity: adherence, quality of 
program delivery, and participant responsiveness. We found that the measure could be reliably 
utilized by trained observers both during targeted SDLMI instruction (Part B) and during other 
content instruction where the expectation was that concepts associated with the SDLMI would be 
embedded (Part C). The preliminary findings from the use of the tool in the first semester of a 
large RCT confirms the importance at looking at the three dimensions of fidelity as scores varied 
across these dimensions and across three phases of the SDLMI. 

While the findings are informative for ongoing work on the SDLMI, they also raise 
broader issues to consider with the implementation field, particularly in relation to complex 


interventions and supporting coaches and implementers of the SDLMI. First, what is the 
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expected level of fidelity across differing dimensions? Or perhaps, more accurately, what is the 
necessary level of fidelity for influencing student and teacher outcomes and do — as other 
research has suggested — contextual factors influence this? Second, what is the best way to 
ensure the complexity of fidelity is recognized in such a way that we can meaningfully report on 
fidelity to inform analytic models and present to diverse stakeholders during, for example, a 
coaching process? Third, how can information gathered from multiple sources (e.g., teachers, 
coaches, blinded observers) be meaningfully integrated? Are all sources of information needed? 
Does high congruence exist between different raters, and if so, can we rely on teacher or coach 
report under some circumstances? Fourth, how can we account for influencing factors in fidelity 
assessment and implementation? For example, the SDLMI can and has been utilized in other 
contexts including transition planning and adult support services in addition to inclusive, general 
education classrooms. Can the same fidelity framework be utilized if contextual factors are 
documented? How can this information be integrated into analytic models? And, how can and 
should factors that influence fidelity of a complex intervention (e.g., program differentiation, 
training of facilitators) be integrated into models and used to contextualize not only assessed 
fidelity, but the relationships between the intervention, assessed fidelity, and outcomes? Lastly, 
how does fidelity and the factors that influence it change over time? Does fidelity increase or are 
there other patterns and influencing factors that emerge over time? There is a critical need for 
ongoing research that systematically addresses these issues, and collects longitudinal data on 
fidelity and the factors that influence it enabling robust implementation evaluation to further 
delineate the “black box” that has up until this point characterized our understanding of whether 


interventions are delivered and received as intended (Grant et al., 2013). 
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Fidelity Framework and Associated Methods 


Component Definition How it is Assessed 
Study design/ Ensures that a study can test hypotheses in relation to underlying e Intervention closely aligned with theoretical 
Intervention theory. Detailed or specific interventions have been found to be more foundations 
Complexity likely to be implemented with high fidelity— it is also easier to achieve =e Implementation Schedule with clearly defined 


high fidelity of a simple rather than a complex intervention 


Assessing and providing training of treatment providers to ensure that 
they have been training to deliver the intervention to study participants. 
All providers should be trained in the same manner. 


Training Providers 


implementation and evaluation schedule 
Standardized SDLMI training 

SDLMI Coaching Model 

SDLMI Teacher’s Guide, Mini-Lessons, and 
Supplemental Materials 

SDLMI Online Modules 

Teacher Knowledge, Skills, and Use Survey 
SDLMI Professional Development Survey 


Adherence to the 
program/Content/ 
Dose 


Identification of critical elements of an effective program as well as the 
amount of program content received by participants. The active 
ingredients of the intervention include the drug, treatment, skills or 
knowledge that the intervention seeks to deliver to its recipients. 


SDLMI Fidelity Measure and Teacher Fidelity 
Measure 

Implementation Schedules 

School Profile Form 


Quality of program 
delivery/Degree of 
individualization 


Ratings of provider effectiveness which assess the extent to which a 
provider approaches a theoretical ideal in terms of delivering program 
content. This could mean the use of inclusion and exclusion criteria to 
define the sample from the target population to whom the intervention 
is delivered, or modifications of elements of the intervention to suit the 


SDLMI Fidelity Measure and Teacher Fidelity 
Measure 

Coaching Conversation Notes 

Teacher Self-Determination Knowledge, Skills, 
and Use Survey 


needs of the individuals within the sample. 


Participant Ratings of the extent to which participants are engaged by and involved 
responsiveness in the activities and content of the program 

Receipt of Monitoring how the patients are able to understand and perform 
treatment/ treatment-related skills throughout the treatment delivery. This can be 


Enactment of 
treatment skills 


measured through pre- and post-tests, setting achievement objects and 
reviewing intervention homework assignments. This also includes 
monitoring and improving the ability of the patients to perform 
treatment-related skills in real life 

Identifying unique features of different components or programs so that 
these can be reliability differentiated from one another 


Program 
differentiation 


SDLMI Fidelity Measure and Teacher Fidelity 
Measure 

SDLMI Fidelity and Teacher Fidelity Measure 
Self-Determination Inventory: Student Report 
Goal Attainment Scaling — Student Version 
Goal Attainment Scaling — Teacher Version 
Access Survey 


School Profile Form 
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Table 2 


SDLMI Fidelity Measure: Inclusive, General Education Version Sample Items 


Part Dimension of fidelity Example item 
B Adherence Did the teacher meet the targeted Teacher Objective(s)? 
B Quality of Program Delivery Did the teacher support the students to take steps in solving the problem targeted in 


the specific phase (e.g., setting a goal, creating an action plan)? 


B Participant Responsiveness Did the students demonstrate knowledge of their goals and/or the process to 
achieve them? 


C Quality of Program Delivery Did the teacher reference students’ goals, action plans, and evaluation strategies? 
C Participant Responsiveness Did the students respond to teacher’s questions or prompts? 


Note. Part C does not include an adherence dimension as skills targeted in the SDLMI are intended to be embedded in the content 
instruction during this part of the fidelity observation in inclusive, general education settings. 
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Table 3 


Scale Score Descriptive Statistics — Overall 


Scale Raw 
n mean 
B. Adherence 35 ttl 
B. Quality of Program Delivery 35 15.00 
B. Participant Responsiveness 35 9.23 
C. Quality of Program Delivery 35 8.06 
C. Participant Responsiveness 35 3.07 


Note. SD = Standard deviation; Range of scores per scale: Adherence (0-10), Quality of 


SD 
2.24 
4.32 
2.99 
3.93 
3.17 


Program Delivery (0-23), Participant Responsiveness (0-15) 


mean 
77.14 
65.22 
61.52 
40.29 
37.14 


Percent of Maximum 


SD 
22.44 
18.77 
19.94 
19.63 
21.16 


35 
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Table 4 


Scale Score Correlations 


Part B Part C 
Woherencé Quality of Participant Quality of Participant 
~~ delivery = responsiveness delivery responsiveness 
B. Adherence 1 
B. Quality of delivery 0.759 1 
B. Participant responsiveness 0.707 0.865 1 
C. Quality of delivery 0.356 0.512 0.42 iE 
C. Participant responsiveness 0.309* 0.481 0.435 0.835 1 


Note. * = not statistically significant. 
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Table 5 


Scale Score Descriptives — By Phase 


Raw Percent of Maximum 
Phase Scale 
mean sd mean sd 

B. Adherence 7.43 2.41 74.29 24.09 

im B. Quality of Program Delivery 14.64 4.34 63.66 18.88 
8 B. Participant Responsiveness 9.21 2.99 61.43 19.94 
aw C. Quality of Program Delivery 7.50 4.57 37.50 22.85 
C. Participant Responsiveness 9.07 2.95 33.81 19.65 

B. Adherence 8.42 1.44 84.17 14.43 

ey B. Quality of Program Delivery 16.17 3.41 70.29 14.81 
3 B. Participant Responsiveness 9.58 2.94 63.89 19.58 
a C. Quality of Program Delivery 9.00 3.54 45.00 ley a | 
C. Participant Responsiveness 6.67 3.11 44.44 20.76 

B. Adherence 722 2.82 72,22 28.19 

. B. Quality of Program Delivery 14.00 5.41 60.87 23.51 
8 B. Participant Responsiveness 8.78 3.00 98.52 22.31 
ay C. Quality of Program Delivery 7.67 3.50 38.33 17.50 
C. Participant Responsiveness 4.89 3.55 32.59 23.67 


Note. SD = Standard deviation. 


